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WHAT IS CLAIMED IS: 

1 1 . A system, comprising: 

2 a computer configured to determine a position and shape of an object of 

3 interest from video images and characterize activity of said object of interest 

4 based on analysis of changes in said position and said shape over time. 

1 2. The system of claim 1, further comprising: 

2 a video camera coupled to said computer for providing said video images. 

1 3. The system of claim 2, further comprising: 

2 a video digitization unit couple to said video camera and said computer 

3 for converting said video images provided by said video camera from analog to 

4 digital format. 

1 4. The system of claim 3, further comprising: 

2 a storage/retrieval unit coupled to said video digitization unit, said video 

3 camera, and said computer, for storing said video images and standard object 

4 video images. 
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15. The system of claim 1, wherein said computer includes an object 
2 identification and segregation module receiving said video images. 

1 6. The system of claim 5, wherein said object identification and segregation 

2 module operates using a background subtraction algorithm in which a plurality 

3 of said video images are grouped into a set, a standard deviation map of the set 

4 of video images is created, a bounding box where a variation is greater than a 

5 predetermined threshold is remove from said set of video images, and the set of 

6 images less said bounding boxes is averaged to produce a background image. 

1 7. The system of claim 6, wherein said computer further includes a behavior 

2 identification module for characterizing activity of said object, said behavior 

3 identification module being coupled to said object identification and segregation 

4 module. 

1 8. The system of claim 7, wherein said computer further includes an object 

2 tracking module for tracking said object from one frame of said video images to 

3 another frame, and an object shape and location change classifier for classifying 
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4 the activity of said object, coupled to each other, said object identification and 

5 segregation module, and said behavior identification module. 

1 9. The system of claim 8, wherein said computer further includes a standard 

2 object behavior storage module that stores information about known behavior of 

3 a predetermined standard object for comparing the activity of said object, said 

4 standard object behavior storage module being coupled to said behavior 

5 identification module, and a standard object classifier module coupled to said 

6 standard object behavior module. 

1 1 0. The system of claim 5, wherein said computer further includes a standard 

2 object behavior storage module that stores information about known behavior of 

3 a predetermined standard object for comparing the activity of said object, said 

4 standard object behavior storage module being coupled to said behavior 

5 identification module. 

1 11. The system of claim 1 , wherein said object is a living object. 

1 12. The system of claim 1 , wherein said object is an animal. 

52 



r 

f 



PATENT APPLICATION 
Attorney Docket No.258/I60 



1 13. The system of claim 1, wherein said object is a mouse. 

1 14. The system of claim 1, wherein said object is a human. 

1 15. The system of claim 1 , wherein said obj ect is a man made machine. 

1 1/5. A method of determining and characterizing activity of an object using 

2 computer processing of video images, comprising the steps of: 

3 detecting a foreground object of interest in said video images; 

4 tracking changes to said foreground object over a plurality of said video 

5 images; 

6 identifying and classifying said changes to said foreground object; and 

7 characterizing said activity of said foreground object based on comparison 

8 to activity of a standard object. 



9 

1 1 7. The method of claim 1 6, wherein said step of characterizing said activity 

2 includes the steps of: 

3 describing a sequence of postures as behavior primitives; and 
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4 aggregating behavior primitives into actual behavior over a range of 

5 images. 

1 18. The method of claim 16, wherein said foreground object detection 

2 includes the step of generating a background image from an average of a set of 

3 individual frames of said video images. 

1 19. The method of claim 18, wherein said step of generating a background 

2 image includes the step of determining variation in intensity of pixels within said 

3 individual frames to identify a region where said foreground object is located. 

1 20. The method of claim 19, wherein said step of generating a background 

2 image further includes the step of using non- variant pixels of the video images 

3 to generate said background image. 

1 21. The method of claim 20, wherein said step of generating a background 

2 image is performed periodically to correct for changes in background objects and 

3 small movements of a camera capturing said video images. 

1 22. The method of claim 16, wherein said detecting a foreground object 
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2 includes using a background subtraction method comprising the steps of: 

3 multiply^ frames in a neighborhood of current image; 

4 apply a lenient threshold on a difference between a current image and a 

5 background so as to determine a broad region of interest; 

6 classify by intensity various pixels in said region of interest to obtain said 

7 foreground object; and 

8 a Pply edge information to refine contours of said foreground object 

9 jmage. 



1 23. The method of claim 16, wherein said step of detecting said foreground 

2 includes the step of manual identification of foreground objects to be tracked and 

3 characterized. 

1 24. The method of claim 17, wherein said posture determination and 

2 description includes using statistical and contour-based shape information. 

1 25. The method of claim 24, wherein said step of identifying and classifying 

2 changes to said foreground object includes using statistical shape information 

3 selected from the group consisting of: 

4 area of the foreground object; 

5 centroid of the foreground object; 
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6 bounding box and its aspect ratio of the foreground object; 

7 eccentricity of the foreground object; and 

# 

8 a directional orientation of the foreground object relative to an axis as 

9 generated with a Principal Component Analysis. 

1 26. The method of claim 24, wherein said step of identifying and classifying 

2 changes to said foreground object uses contour-based shape information selected 

3 fcom the group consisting of b-spline representation, convex hull representation, 

4 and comer points. 

/ 

1 27. The method of claim 24, wherein said step of identifying and classifying 

2 changes to said foreground object includes identifying a set of model postures and 

3 their description information, said set of model postures including horizontal 

4 posture, vertical posture, eating posture, or sleeping posture. 

1 28. The method of claim 27, wherein said step of identifying and classifying 

2 changes to said foreground object includes classifying the statistical and contour- 

3 based shape information from a current image to assign a best-matched posture. 
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1 29. The method of claim 1 7, wherein the said step of describing said behavior 

2 primitives includes the step of identifying patterns of postures over a sequence of 

3 images. 

1 30. The method of claim 29, wherein said step of describing said behavior 

2 primitives step further includes the step of analyzing temporal information 

3 selected from the group consisting of direction and magnitude of movement of 

4 the centroid, increase and decrease of the eccentricity, increase and decrease of 

5 the area, increase and decrease of the aspect ratio of the bounding box, change in 

6 the b-spline representation points, change in the convex hull points, and direction 

7 and magnitude of corner points. 



1 31. The method of claim 29, wherein the step of describing said behavior 

2 primitives step includes behavior of a standard object such as stationary, moving 

3 for left to right and vice versa, standing up, and falling down. 



1 32. The method of claim 29, wherein the step of describing said behavior 

2 primitives step includes a step for providing a means for entering user defined 

3 customized behavior primitives. 
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1 33. The method of claim 17, wherein the said step of determining actual 

2 behavior by aggregating behavior primitives includes the step of analyzing 

3 temporal ordering of the primitives, such as using information about a transition 

4 from a previous behavior primitive to a next behavior primitive. 

1 34. The method of claim 33, wherein said temporal analysis is a time-series 

2 analysis such as Hidden Markov Model (HMMs). 

1 35. The method of claim 33, wherein the said step of determining actual 

2 behavior includes identifying actual behavior selected from a group consisting of 

3 sleeping, eating, roaming around, grooming, and climbing. 

1 36. A method for background subtraction of a video image, comprising the 

2 steps of: 

3 grouping a number of images into a set of video images; 

4 creating a standard deviation map of the grouped images; 

5 removing a bounding box area of said image where variation is above a 

6 predetermined threshold to create a partial image; and 
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combining said partial image with an existing set of partial images by 
averaging the set of images to generate a complete background image deplete of 
a desired foreground object. 

37. The method of claim 36, further comprising the step of subtracting said 
complete background image from a current image so as to obtain said desired 
foreground object. 

38. The method of claim 36, wherein said steps are repeated periodically to 
update said complete background image. 

39. A system, comprising: 

a computer configured to detect and characterize at least a single behavior 
of an object of interest based on movement of said object, using video image 
analysis. 

40. The system of claim 39, wherein said object is an animal and said 
behavior is detecting when said animal is freezing or a touch or sniff of a 
particular item. 
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1 41. The system of claim 39, wherein said object is an animal and said 

2 detecting and characterizing said behavior is determined by comparing behavior 

3 of said animal against a predetermined norm. 

1 42. The system of claim 39, wherein said object is an animal and 

2 characterizing said behavior is determined by analyzing a daily pattern of said 

3 object against a statistical norm so as to detect effects of drugs or genetic 

4 manipulations on said anima. 



